Dual sensor system and related data manipulation methods and uses

ABSTRACT

The invention relates to sensor systems and related data manipulation methods and software and use thereof, for instance amongst others, in surveillance systems, e.g. fall detection, more in particular systems and methods for capturing data of a scene is provided, comprising a first sensor, providing a first data set; a second sensor, spatially arranged with respect to the first sensor in a predetermined arrangement, the second sensor providing a second data set; and data manipulation means using said first and/or second data set to support enhanced data computations on one or both of said data sets to generate said scene data.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a national stage application under 35 U.S.C. §371 ofPCT/EP2014/050004, filed on Jan. 2, 2014, which claims the benefit ofEP13150169.4 filed on Jan. 3, 2013, and U.S. Provisional Application61/748,536 filed on Jan. 3, 2013, the contents of each of which areincorporated herein by reference.

FIELD OF THE INVENTION

The invention relates to sensor systems and related data manipulationmethods and software and use thereof, for instance amongst others, insurveillance systems, e.g. fall detection, or other uses in industry.

BACKGROUND OF THE INVENTION

Many (mostly elderly) people accidently fall, when staying alone intheir house/flat/room, and can't recover on their own. This can leavethem helpless for hours, sometimes even days. It can have severeconsequences, and can even result in the death of the person

One has tried for a very long time to find a solution, where a systemwould automatically detect such a fall, and would activate a warning.

The most used system today is a device with a help button: when theperson falls, and can't get up on its own, he/she can activate a buttonthat will warn family/an emergency company or other. However, it hasbeen shown that in real cases, in 4 out of 5 incidents, the persondoes/can't push this button (or unable to do so, or did not carry thedevice).

State of the art today is a system that one carries with them, and thatcontains some inertial sensors (acceleration and/or gyroscopes). Thissystem will detect automatically a fall. If the person can't indicateafter the fall that he/she is OK, it will again release a help request.

The big advantage is that it does not require an action of the person toactivate the help call, but the person must wear the device. Also here,experiences shows that this is often not the case (one forgets to wearit when coming out of bed, after taking a shower, or one intentionallydoes not want to wear it since one does not want to be seen asaid-needed).

Also, since this system only detects the movement, and not the endposition (is the person on the floor, or on a chair, . . . ) it oftentriggers falls alarms.

New investigations are now done on contactless systems, mostly based onnormal visual cameras. Although that his solves the problem of therequirement to carry a device, these systems are not yet reliableenough. Another disadvantage is that a visual camera is perceived as aprivacy intrusion. Trials are now done with dual visual cameras, so thatone can calculate the distance to the person. But this is complicatedand expensive.

Various groups are now working on using a Time of Flight camera, inorder to have reliable distance data to distinguish objects. Even thoughit already gives more reliable results, the nr of falls alarms is stilltoo high.

Another approach tried out is the use of PIR (pyro electrical passiveinfrared sensors). These are the type of sensors are also used forburglar alarms. Initially one used single devices, but nowadays thereare also array sensors on the market. They detect movements of a warmobject. But since these are AC sensors, they can't detect a warm objectif it is standing still. Therefor one also has tried to combine thistype of sensors with pressure sensors, in order to have more informationabout the position of the object.

Other attempts to use temperature as detection mean have used antemperature it sensor, to evaluate the situation, triggered by apressure sensor, after the fall (design for fall detection system withfloor pressure and infrared image). But the fall detection by pressuresensors is very limited. Also evaluating the final situation only bytemperature is prone to errors. Another approach uses a temperaturearray to detect a fall from a toilet. Since it only relies ontemperature, it is also very prone to the errors, since a very smalltemperature window is used to detect the position.

Till today, camera systems are not mature enough and therefore there areno fixed mounted systems on the fall detection market.

In the art sensor fusion, being the combining of sensory data or dataderived from sensory data from disparate sources such that the resultinginformation is in some sense better than would be possible when thesesources were used individually, wherein the term better in this case canmean more accurate, more complete, or more dependable, or refer to theresult of an emerging view, is known but still does not providesufficient performance.

Bringing data set of a different resolution to a same resolution by up-or down sampling one of the sets is known but does not alterperformance.

Use in detection algorithms of multiple data sets of different nature ingeneral is known (Julien Ros et al—A generative model for 3D sensors inthe Bayesian Occupancy filter framework: Application for fusion in smarthome monitoring, INFORMATION FUSION, 2012, 13^(th) Conference), inparticular the use of a first detection algorithm on a first set and usethe result thereof as input for a second algorithm operable on a secondset, but such technique still does not provide sufficient performance,nor do they improve the data sets quality itself.

AIM OF THE INVENTION

The aim of the invention is to solve the above mentioned shortcomings ofthe state of the art technologies, in particular the fact that they arenot capable to extract the right conclusion on scene data, and forinstance in case of a fall detection use case generate warnings when notneeded (False Positive), and no warning in case of a real fall (FalseNegative). A further aim of the invention is to provide data capturingsystems for industrial applications wherein enhanced temperature ordepth data is required

SUMMARY OF THE INVENTION

The invention relates to systems, methods, computer program products andrelated storage media and uses as described in any of the claimsappended hereto.

In a first aspect of the invention a system for capturing data of ascene is provided, comprising a first sensor, providing a first dataset; a second sensor, spatially arranged with respect to the firstsensor in a predetermined arrangement, the second sensor providing asecond data set; and data manipulation means using said first and/orsecond data set to support enhanced data computations on one or both ofsaid data sets to generate said scene data, more in particular saidfirst and second sensors are based on different data capturingprinciples; and/or (typically as a consequence of such differentprinciples) said first and second sensors have a different resolution.The invention exploits precisely this different nature of the sensors toenhance the information content in one or more of those data sets andpreferably bring the data sets to a same or substantially the sameresolution.

In a first embodiment of the invention said first sensor is a passivesensor, more preferably said first sensor is a temperature device, evenmore preferably an absolute temperature measuring or sensing deviceand/or a DC temperature measuring or sensing device (meaning capable toread a constant temperature). In an example thereof said first sensor isbased on (an array of) a (IR) thermopiles. In an alternative embodimentsaid first sensor is based on (an array of) a bolometer. Also morecomplicated arrangement comprising both examples is possible.

In a second embodiment of the invention said second sensor is an activesensor, more preferably said second sensor is a distance device. As anexample said second sensor is a (an array of) TOF device. The first andsecond embodiment of the invention can be combined.

In a further embodiment of the invention the data manipulation meansuses said first and second data set in accordance with a pre-establishedspatial relationship of corresponding positions in the scene (be itdirectly or after correcting for the spatial arrangement of the bothsensors with respect to each other), hence the means is able to linkdata in the memory of the data manipulation means based on thecorresponding scene positions.

In a further embodiment of the invention the data manipulation means isable to distinguish a plurality of objects in the scene. The inventionthereof enables an object based approach, of instance said adapting ofpart of said first data set can be based on recalculating part of saiddata set by assigning contributions in the first data set to objects,said assigning being based on said second data set (such asdistinguishing whether an object is part of the foreground or backgroundof a scene), more preferably said recalculating takes into accountproperties of objects. The invention therefore exploits allcomplementary information of both data sets where one data set relatesto a feature of an object per se while the other data set relates torelative aspects of objects with respect to each other. This incombination with object characteristics allow to perform datamanipulation based on the underlying physics in the scene or at least anapproximate model thereof.

The ‘enhanced data’ computations are defined as to enhance theinformation content in at least one of those data sets by use of theother of said data sets, having a different nature. Note that adaptingthe resolution by classic up- or down sampling may be used incombination with the invention, which in itself however does not enhancethe information content nor is it based on the other data set (exceptfor knowledge of its resolution), nor is the different nature thereofexploited. Similarly sensor fusion can be used, but the invention goesbeyond the mere combining of sensory data, as part of one of the datasets is actually changed by use of the other data set, to enhance thisdata. Therefore the data manipulation means uses at least one of saidfirst and second data sets to support enhanced data or data enhancementcomputations on at least the other one of said data sets, as part of orto further generate said scene data, more in particular on pixel level.Therefore one adapts part of said first data set (itself) by use of saidsecond data set (or otherwise). Otherwise stated the adapting of part ofsaid first data set is based on recalculating part of said data set byassigning contributions in the first data set, said assigning beingbased on said second data set. In a data fusion no such recalculation byassigning contribution is performed based on another data set, at mostcopying of data in the up-sampling, to get to the same resolution, isperformed, but no contribution assignment based on the other data set.Note that the data manipulation is based on the underlying physics inthe scene or at least an approximate model thereof and therefore goesbeyond combining and/or up- and down-sampling.

In a further embodiment said data manipulation means is able to detectmovement of a plurality of objects by performing object tracking on the(adapted) first and/or second data set, more in particular saiddistinguishing of objects is based on said tracking. Hence the inventionrecognizes the need of historical data for reliable object recognition.

Finally in a further system embodiment the system provides support forfall detection as a use case, more in particular said data manipulationmeans is capable of detecting a fall of an object (e.g. a selectedobject as a warm object) in the scene based on the data of the scene,even more preferably said data manipulation means takes into account thefinal position of the object, to avoid false fall detection alarms.

In a second aspect of the invention methods for capturing data of ascene are provided, comprising inputting a first data set of a firstsensor; inputting a second data set of a second sensor; and performingdata manipulations using said first and second data set to supportenhanced data computations on one or both of said data sets to generatesaid scene data. Note that the execution of said methods can be on anyhardware either dedicated hardware or general purpose hardware (CPU,GPU) or combinations thereof. Further the delivery of the data from thesensors to the data manipulation hardware can be wired or wireless. Theproposed methods support any of the data manipulation steps describedbefore. Also the generated alarms in case of a fall detection can begenerated locally (light, alarm signal) in the system and/orcommunicated to other surveillance systems in a wired or wirelessfashion.

In an embodiment of this aspect of the invention the step of datamanipulation adapts part of said first data set by use of said seconddata set.

In a further embodiment thereof the step of data manipulation uses saidfirst and second data set in accordance with a pre-established spatialrelationship of corresponding positions in the scene.

In yet a further embodiment the step of data manipulation performs thestep of distinguishing a plurality of objects.

In further embodiments said adapting of part of said first data set isbased on recalculating part of said data set by assigning contributionsin the first data set to objects, said assigning being based on saidsecond data set, preferably said recalculating takes into accountproperties of said objects.

In a further realization thereof said step of data manipulation performsa step of detecting of movement of a plurality of objects by performingobject tracking in the (adapted) first and/or second data set.

In a more preferred implementation said distinguishing of objects isbeing based on said tracking.

Further a method is provided for detecting a fall of an object in ascene, comprising executing any of the methods discussed above; anddetermining a fall of an object in the scene based on the data of thescene, where preferably said determining takes into account the finalposition of the object.

In a further embodiment the above methods further comprise of datamanipulations for performing data corrections taking into account thedifference in angle of incidence of the scene data on the first andsecond sensor.

In a third aspect of the invention a computer program product isprovided comprising code segments which when executed on a suitableprocessing engine implement those steps in any of the above discussedabove.

In a fourth aspect of the invention a machine readable signal storagemedium is provided, storing the computer program product, providedabove.

In a fifth aspect of the invention a use is provided of any of thesystems, methods, computer program products or machine readable signalstorage medium described above for tracking of a living object and/ordetecting the fall of a living object.

In a sixth aspect of the invention a further (calibration) method isprovided of loading in the systems discussed before a predeterminedspatial relationship of corresponding positions in the scene in thefirst and second data set.

In a seventh aspect of the invention a database of objects is provided,suitable for use with and accessible by computer program products asdescribed before, comprising: a plurality of objects with at least thetemperature of an object and its temperature dynamics properties asattributes; and rules for adapting said attribute related to temperaturebased on said temperature dynamics properties and distance informationreceived via said computer program products.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a typical positioning of the sensor cluster.

FIG. 2 shows an example of TOF sensor picture.

FIG. 3 illustrates the temperature sensing approach in accordance withthe IR principle.

FIG. 4 provides a FOV detail of 1 IR pixel.

FIGS. 5A and 5B show a FOV example of IR sensor array and the pixelposition in the whole FOV. The array consists of 64 IR sensors (alsocalled pixels). Each pixel is identified with its row and columnposition as Pix(i,j) where i is its row number (from 0 to 3) and j isits column number (from 0 to 15).

FIG. 6 provides an example of a mechanical construction of theinvention.

FIG. 7 show a realization of the methods steps, here calibration andfurther algorithm.

FIG. 8 shows the mapping of IR signals onto TOF resolution to therebyimprove the temperature data in accordance with the invention.

FIGS. 9A, 9B and 9C demonstrate an object tracking improvement in timeexample of a hand moving away from sensor.

FIG. 10 illustrates the operation of the methods for detecting a personfalling against cupboard.

FIG. 11 the operation of the methods for detecting person in a chair intime.

FIG. 12 shows a more detailed flowchart of an embodiment of theinvention.

FIG. 13 shows the concept of improving depth data.

FIG. 14 shows the various data sets used in accordance with theinvention.

FIG. 15 shows an experimental scene to illustrate the performance of theinvention.

FIG. 16 shows the result obtained by various embodiments of theinvention, compared with a prior-art method.

FIG. 17 shows a simplified temperature splitting example as used by oneof the embodiments of the invention.

FIG. 18 shows an example of bringing the temperature resolution towardsTOF resolution in accordance with an embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

The invented systems and methods are based on distance and temperaturemeasurements, optionally provided in an array format and, more inparticular provide possibilities to extract scene information such as afall of an object in such scene (without displaying the entire scene),and do this by intelligently combining (such as mapping) the relateddata sets, based on one or more of the following approaches: datalinking based on scene position; applying (a model of) the physics inthe scene such as applying (a model of) the physics of objects in thescene; using historical or temporal data (via object recognition and/ortracking and/or corrections in time based on object characteristics). Inan embodiment of the invention intelligent combining of 2 technologiesis provided, more in particular distance via Time-of-flight sensors andanother scene property (for instance infrared thermopile sensors).

Time-of-flight is a technology where one sends out light, and measuresthe time that it takes for this light to bounce to an object, and returnto the sensor. For every meter distance, it will take 6.6 nsec toreturn. With an array of these sensors, one can measure the distance inevery direction within a field of view. A typical resolution nowadays is140 pixels×160 pixels. Higher resolutions are now coming on the market.With this technology one can detect objects/persons, and see where theyare and how they move. This distance information increases drasticallythe reliability, compared to normal visual cameras. In comparison toinertial sensors, it does not only detect movement, but also the finalstatic situation (it can see that the person is indeed lying on thefloor). Despite of the benefits of this type of sensor, it can't providea fault proof detection. Other objects can move and fall on the ground(for instance a coat, or a chair that falls, . . . ). Also the objecttracking itself is not flawless, since it is not always obvious todistinguish the different objects in a picture. FIG. 2 shows an exampleof TOF sensor picture.

This invention therefore combines the system with a temperaturemeasurement, more in particular a passive DC infrared sensor array. Thissensor can measure the absolute temperature of an object/or person atdistance. Every object emits or absorbs energy to its environment asdescribed in the Stefan-Boltzmann relationship: P=e*σ*A*(To⁴−Tc⁴), wheree=the emissivity factor, σ=5.67E-8 W/m²K⁴, A=area of the object,To=temperature of the object and Tc=temperature of the surroundings.

Simplified one can say that an object will radiate about 6 W/m2 per degC. difference with its environment when the object can be considered asan it black body. Most body parts and clothing will be approaching anideal black body. The sensor will collect a small portion of this energy(the portion of the emitted energy which falls on the sensor area). Whenusing the thermopile sensor, then this energy will heat up a membrane(typically with a few mdeg C.), and a series of thermocouples willtranslate this temperature difference of the sensor membrane and therest of the sensor device, into a voltage. A microprocessor (orequivalent) will then calculate the object temperature based on thisvoltage and the temperature of the sensor itself. It also can do this inevery direction within a field of view. For a thermopile array thetypical resolutions are smaller (for instance 16×16 pixels per sensor.).The proposed technology, a thermopile array, allows to measure absolutetemperatures in a simple and economical way. Other inexpensive IRsensors can detect only movement in temperature (pyro-electric, alsocalled PIR sensors, as used in the older alarms systems), but this againlacks the advantage that one can detect static situations. Analternative infra technology that can measure absolute temperature isbolometers. With this technology, one can increase the resolutions (forinstance 512×512 pixels). FIG. 3 demonstrates temperature sensing inaccordance with the IR principle, whereby an object (20) transmitsradiated energy (30), captured on a sensor of the array (40) whichprovides data to a microprocessor (or the like) (50) for computing asignal indicative for the mean temperature (60), more in particular themean temperature within the FOV of a pixel. FIG. 4 provides a FOV detailof 1 IR pixel with (100) the ideal pixels response (pixels measures themean of a 3 deg FOV) and (110) the real pixel response (pixel measuresenergy over a wider FOV (multiplication with the FOV response)). If anobject (equal in temperature) falls for 100% within the FOV of a pixel,then this pixel will measure exactly the temperature of the object. Inthis example the object is then within the 10 deg FOV covered by thepixel; hence, for smaller temperature objects, we need additional data,to calculate the exact temperature on every point (correlation withobject definition, historical data of the temperature, . . . ). FIGS. 5Aand B show a FOV example of IR sensor array. For instance T037 in FIG.5B shows the response of pixel 37 as a function of the position of aheater on the vertical line in FIG. 5A. As one can see, 1 point in placewill fall within the FOV of typically 10 sensors. So if we would have ahand at 4 meter distance, then instead of 1 pixel showing for instance31 deg C., we would have 10 pixels with temperatures between 20 and 30deg C.

The above selection of sensor technologies based on in depthunderstanding of the needs of the scene surveillance use underconsideration requires in addition suitable data handling or datamanipulation techniques to realize the enhanced analysis of the scene asindeed the different of nature of the sensor data does not allow (norwould result in any use in) mere addition of those.

In a first realization one can use object tracking on the distance databut by knowing now also the absolute temperature of the object, one canmake a distinction between a ‘cold’ object, and a living person, moreover one can easily detect the warm parts of a body (like the head andhands), and hence work on sub object level is necessary, to discriminatethe various objects in the scene. Warm objects can further be dividedinto sub-objects, related to the main warm object (person), but having adifferent temperature.

In a second realization an even more sophisticated algorithm isprovided, based on a full mapping (intelligent combination) of thedistance and temperature. The pixels size of the TOF is mostly smallerthan that of the thermopile pixels, and therefore the special resolutionof the TOF will be very high. The thermopile pixels will have fewerpixels, and the FOV of 1 pixel will be rather large. The mappingalgorithm will map a temperature on every TOF pixel.

For use of the invention system and method in the application of falldetection, the sensor cluster is preferably positioned on the ceiling,possibly in the middle of the room. FIG. 1 (left side—top view of aroom, right side—side view of a room—not on scale) shows a typicalpositioning of the sensor cluster (10) but the invention is not limitedthereto. Both sensors will have a Field of View (FOV), large enough tooverview the full room. A FOV of 160 deg would be advisable. Bothsensors will divide this big FOV in a large number of pixels (small FOVarea's). A typical pixel size for the TOF would be 160×120, having a FOVpro pixel of around 1 deg. This FOV can be achieved with 1 sensor, usingfish eye optics, since the wavelength used is typically around 850 nm(out of the visual spectrum, but still detectable by silicon). The pixelsize for the Thermopile typically will be smaller. The wavelength usedfor passive infrared, is typically between 5 and 10 um. Due to thenature of these wavelength, it is very difficult/expensive to make wideFOV optics, so a possible configuration is using 4 sensors with typical80 deg C. FOV. One sensor could have 32×32 pixels, so 4 sensors wouldcover 64×64 pixels, a 1 pixel having a FOV of around 3 deg C.

The strength of the invention for its applications is in the fusion(combination) of both sensors inputs, on which objects are defined,which in time will become more precise. A typical construction is shownin FIG. 6 (top view on top, side view on the bottom of the figure). Herethe TOF sensor comprises of LEDs or a TOF light source (200) and asensor (210). Further there is a plurality of IR sensors (220), a mirror(230), processor unit (240) and TOF optics (250). Due to the mechanicaltolerances of the chip position in the chip package, the opticsassembly, and the positioning of the sensors on the PCB, there will bequite some variations on the mechanical matching of the ir pixels, ontothe TOF pixels. For this, a calibration cycle, after assembly will bedone.

Since the FOV of an ir pixel is larger than that of TOF pixels, and evenmuch larger than the ir pixels distance, the invention providesalgorithms to be able to map on every TOF pixel, a temperature. Toovercome the uncertainty of this matching which would be present if oneonly considers 1 point in time, the algorithms preferably use also thehistorical information to improve the matching in a manner beyond meretracking.

Objects, as well as ‘the background’ will be defined. The background iswhat is not moving for a certain amount of time AND what is not a warmobject. Objects are items which are moving, or have moved somewhere intime. They will be split up into warm and cold objects. Warm object aredefined as ‘living’ items, as a person, a pet, . . . . Cold objects thatdo not move for a predefined time (for instance a chair that fell), willbecome part again of the background. Warm object never will become partof the background. By tracking these objects (which all have their owntemperature distribution), one can map their temperature profile ontothe TOF pixels.

Once this mapping is in place, the fall detection itself gets mucheasier. We indeed know the warm living objects, and can detect a fastdownwards movement of their center of gravity. The system also allows todetect parts of the object (for instance head, hands, . . . ) so thatalso pre-fall movements (waving with arms at the beginning of a fall)can be detected.

An important advantage of the proposed sensors is that they also canevaluate the static status after the fall. The TOF sensor can indeedmeasure the position of the person relative to the ground, of thebackground (for instance chair, bed, . . . ).

The invention provides for a learning system and learning methods,whereby at least two (linked) sensors interact in time to therebycontinuously improve performance by enhancing over time the quality ofobject (or sub-object) definition and/or detection and/or and tracking.This embodiment of the invention hence uses historical data.

This is now more explained in the FIGS. 8 and 9A, B and C. In FIG. 8, westart from some history data. The left side shows the initial IRresolution. The temperature per pixel is influenced by neighborhoodpixels. The right side shows the mapping onto TOF resolution where everysub-object gets assigned 1 temperature. The system knows we have anobject before a background of 20 deg C. (from history) since ifsomewhere in time, there were no objects before the wall (which is partof the background), then some of the pixels only have seen the wall inits full FOV. As such, the temperature of the wall has been measuredvery precisely. The system also has already split up the object in 2.Warm objects can never disappear. If an object has 2 distinct forms(detected by the TOF), for instance the body and the head, and if the IRsensors indicate a difference in temperature, then the object will besplit in 2 sub-objects, that always need to touch each other. Allsub-objects will be followed in time. However, sub-objects can alsomerge in time. For instance an arm, once defined as sub-object, whenresting against the body for a long time (for instance when reading)will become a temperature closer to the body temperature. So after awhile sitting in a chair, both the TOF and the IR sensor data will beclose to each other, and the system can merge again the arm and body to1 sub-object. So the nr of sub-objects defining an object can change intime. However a warm object itself can never disappear, unless whenmoving out of the FOV of the sensors. We assume in this example that inthis point of time, the system tracks the head and the body as 2sub-objects. The algorithm now will map the energy detected on all ofthe IR pixels, onto the 3 split objects (background, head and body). Itwill use a best fit algorithm to do this. All the pixels with thedistance to the background (wall) will be assigned 20 deg C. It has nowto divide the received energy info over 2 objects. It will for instanceassign based on this 31 deg C. to the head, and 23 deg C. to the body.In case the system has ‘more reliable’ information of the past, it cantake this into account (for instance and decide that the body=24 degC.). In essence the temperature data is split in depth neighborhoodclass temperatures.

FIGS. 9A, B and C show an example of the power of tracking withtemperature and FOV. We assume a total FOV of 180 degrees. We assumethat we have 180 TOF pixels in 1 direction=1 degrees FOV per pixel. Weassume that we have 60 IR pixels in 1 direction=3 degrees FOV per pixel.The bottom shows temperature mapped on FOV pixels while the top showsthe pure IR pixel data. If once in history, the hand is close to thesensor, the IR resolution is good enough to calculate the temp of thehand correctly (enough pixels are within the area that sees the hand).The further away the hand is from the sensor, the more every IR pixelwill see the background, so none of the pixels will read the exact handtemperature (since in its FOV it also will see part the backgroundtemperature=20 deg C.). However, since the FOV sensor will give theexact area of the warm object seen by the sensor, it will reassign theenergy received by the sensors, onto the warm object (for instance byuse of an inverse weighted averaging). So the wall will become 20 deg C.(see before), and the hand will be assigned the calculated temperature.As long as the area of the sum of the TOF pixels is close to the area ofthe objects (so enough FOV pixels within the object), the results willbe fairly correct (see hand at 1.5 m). However if we only have a few FOVpixels (see hand at 4 m), then the area of 2 pixels, will deviateconsiderably from the object area, and as such the calculatedtemperature will not be correct. But the system knows the history of thetemperature assignment of the hand, and will adjust this calculatedtemperature (for instance from the calculated 27 deg C. to 29.5 deg C.)for case 3. It will not necessary assign 31.5 deg C., since it will usea weighting of all inputs, to come to a best estimate. This will takeinto account that objects can and will change of temperature withincertain limits: —hand can only change of temperature slowly (when nottouching objects). —the temperature of the head can change depending onthe positions (for instance when standing under the sensor, the systemwill measure the temperature of the head (for instance 29 deg C.), whileif walking away from the sensor, the system will see the face itself(for instance 31.5 deg C.). It is important to acknowledge that thesensors measure the surface temperature. So short hair will give acloser temperature to the skin temperature than very curly hair. Thesurface temperature of a hand will cool down more when there is a windgoing through the room (or equivalent, when the hand moves in relationto the air).

FIG. 10 shows the advantage of the temperature mapping. When using apure FOV sensor, and when a person runs into a cupboard, the system canget confused. If now the cupboard, or the person falls, the system willnot know for sure whether there is a person falling. By tracking thetemperature, we can easily make this decision. Alternatively FIG. 11shows that when a person goes to sit in a chair, a pure TOF sensor willbe able to track this. However, when the person is sitting still in thechair for a long period, the TOF can lose track: due to noise on thepixels, sun shining on the person, . . . the imprecision on the distancecan become so high, that the system will lose the person. Again, theproposed system will keep track of the ‘warm object’, and can easily dothis by tracking the head with its temperature sensor.

FIG. 12 shows a flowchart of an embodiment of the invention. Onerecognizes a step of calibrating (1). The assembly of the IR sensor andthe TOF sensor itself will have deviations. Also the assembly of thesensors in relation to each other, including the optics can have furtherdeviations. This will be corrected with an initial calibration step. Onerecognizes another step of object creation (2). This routine willcontinuously monitor the TOF data variations, within a predefinedtiming. When there is a movement, not linked with a predefined object, anew object is created. This can be due to a movement within the room(for instance fallen chair), or a person entering the room. The systemhas learned the most common entry and leaving positions of warm objects(for instance a door) and this is stored in the attribute database. Thisis then again an input for supporting the object creation module. Once anew object is detected, an IR image map will be used to classify theobject into a cold or a warm object. If a warm object, possibly alreadysub-objects will be defined. The end results (object definition andtemperature) will be stored into the object database. Yet another stepis object tracking and updating (3). This routine will continuouslyinput new IR and TOF array data. It will first verify if the existingwarm sub-objects are still valid, or if some have to be added or merged.It also will check whether cold objects are still valid (for instancewhen not moved for a predefined time). Parameters for these decisionswill be taken from the attribute database. Once the (sub) objects arefixed, then the IR array data will be used to map a temperature on everyof the (sub) objects. The final (sub) objects position and temperaturewill be stored in the object database. Yet another step is movementdetection (4). This routine will continuously monitor the movement ofthe warm objects and sub-objects, as available in the object database.It will compare this with the rules stored in the object attributesdatabase. Outcome of this routine is whether or not a fall movement ishappening. A further step is fall verification (5). When a fall movementhas been detected, this module will check the end situation (position ofthe warm object in relation to the room mapping—background), and basedon the rules defined in the attribute database, it will conclude whetheror not we have a valid fall event (for instance a person sitting fast inthe chair will be seen as a fall movement, but not classified as afall). Another step is pre-warning (6). When a fall has been detected,then the system will check if the person can recover (for instance bystanding up again), or if the alarm can be stopped (for instance by ahuman intervention, when for instance another person in the room didfall, but does not need external support). Another step (7) is to startan alarm. When no recovering or stopping actions occur within apredefined time (attribute database) then an alarm will be given. Insummary various steps of a method are disclosed which can be implementedeither separately or combined. Some steps are initial steps (forinstance the calibration step) done once or once in a while (forinstance to cover for small changes of the setting over time). Somesteps are run in parallel (for instance the object creation, objecttracking and updating and the fall movement detection). Besides themethod, the invention discloses a system or an assembly (sensors, opticsand other electronics) and related software or supporting softwarestructures as a specialized object database.

Note that FIG. 7 shows a realization of the methods steps, herecalibration and further application of the algorithm. A step (300) ofmechanical matching the IR and TOF sensor is identified. Further a step(310) of initial mapping of IR and TOF pixels. Further a step (320) ofdefining and tracking of object or sub objects in time. The next step(330) provides the updating of the IR mapping to TOF pixels. Step (340)provides the fall detection (when a warm object moves towards thefloor), followed by verification (350) of the final position of theperson. Finally an alarm is started in step (360). Note the continuousloop between steps (320) and (330).

Further the combined data approach also allows for active steering ofthe active sensor, for instance providing more light into zones wheredue to external circumstances such as sunshine (detectable by thetemperature sensor) the performance of the TOF sensor is less reliable.Indeed a potential problem of a pure TOF system is that the pixels cansaturate due to direct sunshine in the room. This can be overcome byreducing the integration time of the pixel, but this has as consequencethat one has to apply a very small duty cycle on the lightning system(all light energy has to be released in for instance 5% of the time).This will have a negative impact on the reliability and/or the cost ofthe lightning system of the TOF. Now, by using the IR array data, andthe implemented algorithms, one can also tackle this problem: Ifsaturation would occur, then in 1^(st) instance, the system would relyon the IR array for tracking. Whenever the system would need moreadditional info (for instance 1 time per second, or more in case of anevent), then the TOF duty cycle would be changed to tackle thesaturation effect. The same will happen after the fall, since only theTOF can reliably verify the final position of the person. Since we mightuse different LEDs for different FOV areas, one can decide to onlyactivate 1 led in the reduced duty cycle mode. With this system, thereduced duty cycle mode would be used very selectively (only those LEDswhich are required), and only when needed. As such, the lifetimereduction effect on the lightning system can be limited.

The invention relates to static and/or dynamic improving of atemperature array (and mutatis mutandis a distance or depth array). Thetemperature array can be improved using the distance info. The strengthof this innovative technique can be shown by comparing performance basedon an experiment. In FIG. 15 a scene describing 5 objects on abackground is shown, with data expressing the distance in meter, otherdata expressing the measured temperature in deg C., on a lowerthermopile resolution to be compared with the ‘real’ temperature.Various common techniques like nearest neighborhood interpolation,bilinear interpolation or cubic interpolation used to upscale an irarray, using the standard interpolation techniques, lead to deviationsthat can easily be 5 deg C. and more, and mean errors of respectively0.78, 0.83 or 0.80. The temperature improvement that can be obtained, byusing the pixel depth information, in accordance with the invention, isnow described. For every depth pixel (so on TOF resolution), atemperature is calculated, based on the depth information, andoptionally also the confidence of this data, of the surroundingtemperature pixels. Consider the result for the following experiments A,B and C in FIG. 16, compared to the prior-art technique result shown inthe top left corner of FIG. 16. In a first experiment A an interpolationis used, where the weighting factors are based on the depth information(and optional on its confidence level). In example B and C, thetemperature pro original temperature array pixel, is split up towards 2,3 or more temperatures, linked to 2, 3 or more mean height levels,measured (by the TOF sensor) within this original ir array pixel. Thiscan be done with or without maintaining the power budget. This powerfulmethod will allow to recalculate temperatures towards its correct value,which can be lower or higher then all original measured values. In anembodiment of the invention this is done by performing an inverseweighted averaging, meaning by solving an equation wherein thetemperature to be determined is part of the weighted averaging part ofan equation while the outcome of the weighted average is known. In B, analgorithm is used which does not need prior segmentation. In C, priorsegmentation info is taken into account. Those experiments show that hetemperature error pro pixel drastically can be decreased (mean error of0.21 for experiment A), while for methods B and C (mean error of 0.09for experiment B and 0.087 for C), one can also correctly measure thetemperature of objects smaller than the original temperaturemeasurements (on lower spatial resolution). In this example overalltemperature deviation is reduced with a factor of 10. A furtherimprovement is to process several iterations of the above process on thesame static data to further enhance the temperature data. A furthermajor improvement in the recalculation of the temperature and distancemap in accordance with the invention can be achieved by additionallytaking the time information into account. The recalculated confidencelevels assigned to distance and temperature do not only rely on the infoon 1 temperature and 1 distance map, but can take into account theaccumulated confidence data. A wall which has a fixed distance, and onlya slow changing temperature, will gradually get assigned very highconfidence levels. These will then be used to more correctly calculatethe temperature on objects moving before this wall. Another factor ofimprovement can be achieved by taking into account object attributes andrules. For example, when tracking a person, one has learned in time itsheight & volume (attributes), while for the change in time of thetemperature, one will use predefined rules (max change of temperature infunction of object movement and position, air flow predictions—based ongradients in wall temperature—, . . . ).As discussed above the depth array can be improved using the temperatureinfo. An example can be found in the FIG. 13, where we show a part of asolid wall at 4 meter. The confidence of the depth in the wall area A isvery high, while the depth info at 2 parts of the wall is veryunreliable: a part (A) of the view has very low confidence (low lightreflection) due to a black painting on the wall; a part (C) of the depthsensor view is overexposed due to direct falling sunlight. In bothcases, the confidence level of the depth data of these 2 parts is 0(item C), or very low (item A), while the confidence level of thetemperature info is very high. Since from history (time info) we knowthat we do not have any other objects around, based on the equaltemperature info, and we see no important temperature steps in this partof the FOV, we assign the A-depth to area's B and C, with an increasedconfidence level.Generally stated the data manipulation means adapts part of said seconddata set by use of said first data set, wherein said adapting of part ofsaid second data set is based on classifying pixels in said first dataset in a plurality of temperature neighborhood classes based on saidfirst data set; determining confidence information (on the depth valuesbut also for instance on the classification); adapting second data setpixels in the temperature neighborhood class with the high confidence;and recalculating part of said data set in said second data set in oneof the other temperature neighborhood classes with a lower confidence byuse of said adapted second data set pixels.FIG. 17 shows a further explanation of one of the embodiments of theinvention. Assume that 1 ir pixel corresponds with 100 TOF pixels. The 9ir pixels shown see a ground floor at 20 deg C. at 2 meter (afterdistortion correction), but the middle pixel also contains a solidplate. This solid plate at 1 meter has a diameter of 18 cm. All outer irpixels measure approximate 20 deg C. being the wall temperature. Themiddle pixel will see: 90% of its FOV at 20 deg C. and 10% of its FOV at50 deg C. When we simplify the energy radiation laws to a linearequation (6 W/m2/deg C.), the sensor will measure: (1*50+9*20)/10=23 degC. So this is the only temperature information we will receive from thetemperature sensor. From the depth class algorithm, we know that wemainly have 2 depth clusters (depth neighborhood class): at 1 meter andat 2 meter (mean distance 1.9 m). We now have 2 unknowns (temperaturesat 1 meter, and at 2 meter), and only 1 equation (the energy sent by100% FOV at 23 deg C., should be the same as the sum of the energiessent out by the 2 defined objects). This missing information is providedby the temperature of the other ir pixels, but weighted to thedifference in depth to the pixel under splitting. In this simplifiedcase, we see that we have 8 other pixels at approximately the sameheight (2 m), and with only 1 depth (see depth distribution), so these 8temperature pixels have a high confidence. We assign now the sametemperature of 20 deg C. to the 2 m distance depth class. Now that weknow this temperature, we can calculate the temperature at 1 meter) fromthe power equation: (1*t_1 m+9*20)/10=23 deg C., and we will find thatt_1 m=−50 deg C. as the temperature of the small object. This method iscalled the inverse weighted average. Note that this is a rather idealsituation, where the nr of depth classes=2, and the confidence of theone of the 2 depth class temperatures is approximate 100% (since 8 otherpixels with high confidence are on approximately the same height). Thesame reasoning will apply when we have 3, 4 of more depth classes. Inthis case, we will have 1 (power) equation, and 3, 4 or more unknowns.The unknown temperatures will be calculated based on the informationavailable in the other ir pixels, taking into account their confidencelevel (based on their depth distribution, and possible historicalinformation) and the difference in depth towards the pixel undersplitting. Note: Instead of using pure depth difference as input forweighting the contribution of the other ir pixels, one also can useprior segmentation information. The ir pixels within the same segmentedobject, will receive a high weight factor to calculate the temperatureof the distance class corresponding with this (sub) object.FIG. 18 demonstrates that after having improved the original temperaturedata, we can more precisely assign a temperature to every depth pixel.In this example, we have 100 times more TOF pixels than IR pixels. Forevery TOF pixel, a temperature will be calculated from the available irtemperature pixels, but after the splitting step. In this example, wehave 8 original ir temperatures plus 2 splitted ir temperatures, so 10in total, all linked with their depth. The temperature, linked to everyTOF pixel, will use a weighted average of these surrounding ir pixels.The contribution of every of these 10 ir pixels, will depend on theinverse of the height distance between this pixel and the TOF pixel.Before the splitting, the max available temperature (of the 9 pixels)was 23 deg C., after splitting the max is 50 deg C. As can be seen inthe figure, now all TOF pixels can be assigned with the correcttemperature, even if this is higher than the max temperature of the irsensor:

-   -   the TOF pixel (510) looking to the floor has a depth of 2 meter.        It will see 9 original ir pixels with approximately the same        height. Hence the temperature corresponding with this TOF pixel        (510) will be calculated to approximately 20 deg C.    -   the TOF pixel (500) looking to the plate has a depth of 1 meter.        Hence it will see only 1 ir pixel (part of the splitted ir        pixel) with approximately the same height. Hence the temperature        corresponding with this TOF pixel (500) will be calculated to        approximately 50 deg C. Note that this example is a simplified        one. Please note that during the splitting step, we can create        temperatures higher or lower than the original ones. During the        step where we assign temperatures to every TOF pixel, we use a        weighted average of the newly (splitted) ir pixels.        In the above one recognizes that as part of said adapting of        part of said first data set the original pixel temperature data        in said first data set is split in a plurality of depth        neighborhood class temperatures based on said second data set.        With a depth neighborhood class is meant a set of positions in        the scene and the corresponding pixels being at more or less the        same depth (and hence presumably belong to the same context like        an object). Further one observes that a step of determining        confidence information of the temperature of pixels within said        depth neighborhood classes; and use of said confidence        information for said adapting of part of said first data set. In        one particular embodiment said adapting of part of said first        data set is based on selecting first data set pixels in the        depth neighborhood class with a first confidence level,        preferably a high, confidence level; and recalculating part of        said data set in said first data set in one of the other depth        neighborhood classes with a second confidence level by use of        said selected first data set pixels, whereby said adapting of        part of said first data set in one of the depth neighborhood        classes with a second confidence is based on recalculating part        of said data set by an (inverse weighted) averaging of said        selected first data set pixels and the first data set pixels        under recalculation, the inverse weighting is based on the        distance as seen in the second data set.        More generally speaking the invention demonstrates use of data        of different nature (temperature and depth or distance) with a        difference in resolution and/or difference in FOV, wherein in        the methods for data enhancement, various insights and physical        laws are used, such as use of energy radiations laws, to create        a sufficient set of equations. One of those reflects that one        sensor sees a combination of underlying temperatures. The        insights in the scene under study can be obtained by analysis of        the other data set of distances, and providing missing        information by use of the temperature of the other ir pixels,        whereby within such use again a variety of information is used        such as again the depth and/or confidence information and/or        further scene knowledge like assigning of data to objects and/or        the belonging to a depth neighborhood class. In a preferred        embodiment the depth information is used as part of a further        weighting procedure.        Alternatively said the temperature information content in the        original temperature array will be increased by splitting the        original mean temperature per ir array pixel into 2, 3 or more        temperatures linked to depth neighborhood classed (as defined        above). Said depth neighborhood class determination can be        assisted by object recognition or vice versa. This increased        temperature information per ir pixel will be calculated from the        other ir array pixel temperatures, knowing their mean depth,        while maintaining the power budget        (the sum of the energy of the n new temp=equal to the one of the        original temp) In order to apply the correct weight factors to        these other ir pixel temperatures, a weight factor based on the        distance between these other pixels, and the pixel under        modification is used and/or the confidence of the other pixel        temperature is taken into account

In summary the provided system (sensors and data processing means withadapted methods) support applications where both a dynamic and staticevaluation of the sensed scene is advisable for reliable detectionperformance. More in particular the choice that one sensor related to anobject (or sub-object or parts thereof) properties enhances trackingand/or introduction of application intelligence via typical object orsub-object behavior (see waving in fall detection as example) orcharacteristics (for instance temperature distribution based or relyingon the temperature dynamics of such (sub-)objects). Moreover the dualsensing enables logic exchange of the information context of the relatedsignals, by defining notions of background in terms of both signals, bydefining objects in categories also based on both signals. One can statethat the data manipulation results in defining such concepts for furtheruse in an application.

Moreover said data manipulation means uses at least one of said firstand second data sets to support enhanced data computations on at leastthe other one of said data sets to generate said scene data. Moreover insaid enhanced data computations one or more individual pixel values arechanged. Further while described in more detail herein for the casewherein temperature data is adapted or recalculated based on distancedata, the opposite case is equally possible and even a combination ofthose. Indeed one of the embodiments actually stresses the performanceof various loops, going from temperature data to distance data and viceversa, even go along the time axis, by taking into account past data(e.g. a mere averaging out or even also here using confidence basedweighting of such data) and/or even a loop back and forward to higherabstraction levels such as objects, and using additional informationabout those such as their temperature dynamic properties or even otherproperties (speed of movement). Also the loops around the variousmethods in accordance with embodiments of the invention can be used,e.g. weighted and inverse weighted techniques and/or splitting andweighting and/or inverse weighting or combination of all of those. Theinvention exploits the different nature of the two used data sets, inthat for instance in the distance or depth data it may recognizediscontinuities to which it attaches implicit or deduces a physicalreality to correct the temperature data (by making (linear)approximations or more complex computations based on the laws ofradiation, to thereby determine energy contributions). The moving backand forward in the data sets shows to be advantageous in thatre-applying of the methods for distinguishing objects may lead todetection of new objects, not found in previous iterations as they areblurred in the original data. As shown in FIG. 14, starting fromoriginal data of distance and temperature, calculated arrays areconstructed and besides modifying this data in accordance with theinvention (and optionally also including resolution adaptation) alsoincreased corresponding confidence data is constructed. Note that whenusing resolution adaptation this might be performed either on theoriginal data first or as an intermediate step. Hence while said datamanipulation means uses at least one of said first and second data setsto support enhanced data computations on at least the other one of saiddata sets to generate said scene data, in parallel confidenceinformation on one or both of said data sets are created and used insaid data computations. Note that FIG. 14 is an abstract representationof the above concepts. In its implementations one does not necessarilyhave to construe different arrays although this might be one of theimplementations. In accordance with the invention the data manipulationmeans adapts part of said first data set by use of said second data set.In an embodiment thereof said adapting of part of said first data set isbased on recalculating part of said data set by (local) filtering firstdata set pixels, wherein in a further embodiment the filter coefficientis based on said second data set either explicitly or implicitly forinstance by neglecting pixels not considered part of the same depthneighborhood class. In an even further embodiment said adapting of part,or whole, of said first data set (in value) is based on recalculatingpart of said data set by weighted averaging first data set pixels,whereby the weight factors are based on said second data set. The abovedescribes therefore a combination of use of local (neighboring in depth)pixel information with the learning of the physics leading to certaindata from a certain scene.

The invention claimed is:
 1. A system for capturing data of a scene, thesystem comprising: a. a first sensor, measuring temperature, providing afirst data set provided in an array format; b. a second sensor,measuring distance, providing a second data set provided in an arrayformat; and c. data manipulation means using at least said second dataset to support enhanced data computations performed on the first dataset to adapt the first data set by filtering or averaging first data setpixels to generate said scene data; wherein said adapting of part ofsaid first data set is based on recalculating part of said data set byfiltering first data set pixels, wherein as part of said adapting ofpart of said first data set the original pixel temperature data in saidfirst data set is split in a plurality of depth neighborhood classtemperatures, based on said second data set.
 2. The system of claim 1,further comprising determining confidence information of the temperatureof pixels within said depth neighborhood classes; and use of saidconfidence information for said adapting of part of said first data set.3. The system of claim 1, wherein one or more of said adapted first dataset pixels of which one or more is part of said depth neighborhood classtemperatures, is adapted based on recalculating part of said data set byan inverse weighted averaging, the inverse weighting being based on thedistance as seen in the second data set, in combination with filteringfirst data set pixels.
 4. The system of claim 1, wherein said firstsensor is an absolute DC temperature device.
 5. The system of claim 1,wherein said first sensor is based on an array of IR thermopiles and/orsaid first sensor is based on an array of a bolometer.
 6. The system ofclaim 1, wherein said second sensor is an active sensor.
 7. The systemof claim 1, wherein said second sensor is an array of Time-of Flightdevice.
 8. A system for capturing data of a scene, the systemcomprising: a. a first sensor, measuring temperature, providing a firstdata set provided in an array format; b. a second sensor, measuringdistance, providing a second data set provided in an array format; andc. data manipulation means using at least said second data set tosupport enhanced data computations performed on the first data set toadapt the first data set by filtering or averaging first data set pixelsto generate said scene data; wherein said adapting of part of said firstdata set is based on recalculating part of said data set by filteringfirst data set pixels; wherein said adapting of part, or whole, of saidfirst data set is based on recalculating part of said data set byweighted averaging first data set pixels, whereby the weight factors arebased on said second data set; wherein the data manipulation means iscapable to distinguish a plurality of objects, based on said second dataset; wherein said adapting of part of said first data set is based onrecalculating part of said data set by assigning energy contributions inthe first data set to objects, said assigning being based on said seconddata set.
 9. The system of claim 8, wherein said recalculating takesinto account past data.
 10. The system of claim 8 wherein saidrecalculating takes into account temperature dynamic properties ofobjects.
 11. The system of claim 8, wherein said first sensor is anabsolute DC temperature device.
 12. The system of claim 8, wherein saidfirst sensor is based on an array of a IR thermopiles and/or said firstsensor is based on an array of a bolometer.
 13. The system of claim 8,wherein said second sensor is an active sensor.
 14. The system of claim8, wherein said second sensor is an array of Time-of Flight device.